Device for physical interaction between remotely located users

ABSTRACT

An electronic device for touch translation includes a body, a plurality of pins extending from the body, the pins including couplings to facilitate movement of a first portion relative to a second portion of the pins, the pins being controllable to control movement of the first portion relative to the second portion and to control force applied by the pins on an external object. Sensors cooperating with the pins detect forces externally applied to the pins and a communication subsystem is utilized to communicate over a network, with a remote electronic device. A controller is coupled to the pins, the sensors, and the communication subsystem to, based on detected forces externally applied to the pins, transmit a signal to the remote electronic device for the control of the remote electronic device, and to, based on signals received from the remote electronic device, actuate ones of the pins to control the movement of the first portion relative to the second portion and to control the force applied by the pins on the external object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 15/092,507, filed on Apr. 6, 2016, (U.S. Pat. No.10,394,363 to be issued Aug. 27, 2019), which claims priority under 35U.S.C. § 119(e) to provisional application Ser. No. 62/144,283, filedApr. 7, 2015, which applications are expressly incorporated by referenceherein, in their entireties.

FIELD OF TECHNOLOGY

The present disclosure relates to devices for interaction between, forexample, people located remotely from each other.

BACKGROUND

Electronic devices, such as smart phones, tablet computers, laptopcomputers, and desktop computers have gained widespread use for avariety of functions including communications functions. Videocommunication functions, utilizing video chat applications, are commonlyused both for business and for personal use between people locatedremotely from each other, for example, between parents and childrenliving in different locations, between spouses when one or both aretravelling, between colleagues working in different locations, and soforth. Thus, interactions between people are commonly carried outremotely.

Interactions between people utilizing electronic devices forcommunication functions are limited, for example, to voice, video, orboth voice and video communication.

Head-mounted displays may also be utilized for virtual interactionbetween individuals to provide a more realistic interaction. Suchinteractions, however, are only virtual and are limited to interactionin a virtual space.

Improvements in electronic devices to provide further interactioncapabilities between people located remotely from each other aredesirable.

SUMMARY

An electronic device for touch translation is provided. The electronicdevice includes a body, a plurality of pins extending from the body, thepins including couplings to facilitate movement of a first portion ofthe pins relative to a second portion of the pins, the pins beingcontrollable to control the movement of the first portion relative tothe second portion and to control a force applied by the pins on anexternal object, sensors cooperating with the pins to detect forcesexternally applied to the pins, a communication subsystem forcommunication, over a network, with a remote electronic device, and acontroller coupled to the pins, the sensors, and the communicationsubsystem. The controller controls the electronic device to, based ondetected forces externally applied to the pins, transmit a signal to theremote electronic device for the control of the remote electronicdevice, and to, based on signals received from the remote electronicdevice, actuate ones of the pins to control movement of the firstportion relative to the second portion and to control the force appliedby the pins on the external object.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the attached Figures, in which:

FIG. 1 is a block diagram of an example of an electronic device fortouch translation in accordance with one embodiment;

FIG. 2 is a block diagram of an example of a system for touchtranslation including the electronic device of FIG. 1;

FIG. 3A is a perspective view of one example of an electronic device inaccordance with the embodiment of FIG. 1;

FIG. 3B is a partial side view of the electronic device of FIG. 3A;

FIG. 3C is a partial side view of the electronic device of FIG. 3A;

FIG. 3D is a perspective view of another example of an electronic devicein accordance with the embodiment of FIG. 1;

FIG. 3E is a partial side view of the electronic device of FIG. 3D;

FIG. 4 is a flowchart illustrating an example of a method of controllingthe electronic device according to embodiments.

FIG. 5 is a block diagram of another example of an electronic device fortouch translation in accordance with an embodiment;

FIG. 6 is a simplified block diagram illustrating communication betweenelectronic devices, such as the electronic device of FIG. 5, via anetwork; and

FIG. 7A is a perspective view of another example of an electronic devicein accordance with an embodiment;

FIG. 7B is a partial side view of the electronic device of FIG. 7A;

FIG. 7C is a side view of an example of a pin of the electronic deviceof FIG. 7A;

FIG. 7D is a partial side view of the electronic device of FIG. 7A witha portion of a body of the electronic device broken away;

FIG. 7E is a side view of an example of a pin of the electronic deviceof FIG. 7A;

FIG. 7F is a partial side view of the electronic device of FIG. 7A;

FIG. 7G is a partial side view of an example of a pin of the electronicdevice, showing hidden detail;

FIG. 8A is a side view of part of an electronic device in accordancewith another embodiment;

FIG. 8B is a side view of a pin of the electronic device of FIG. 8A,showing additional hidden detail;

FIG. 9 is a flowchart illustrating an example of a method of controllingthe electronic device according to embodiments;

FIG. 10 is a side view of a pin of part of an electronic device inaccordance with another embodiment;

FIG. 11 is a side view of a pin of part of an electronic device inaccordance with another embodiment;

FIG. 12 is a side view of part of yet a further example of an electronicdevice in accordance with yet another embodiment;

FIG. 13 is a perspective view of the electronic device of FIG. 12, withthe pins of the electronic device in the shape of a chair.

DETAILED DESCRIPTION

For simplicity and clarity of illustration, reference numerals may berepeated among the figures to indicate corresponding or analogouselements. Numerous details are set forth to provide an understanding ofthe examples described herein. The examples may be practiced withoutthese details. In other instances, well-known methods, procedures, andcomponents are not described in detail to avoid obscuring the examplesdescribed. The description is not to be considered as limited to thescope of the examples described herein.

The following describes an electronic device and a method for touchtranslation. The electronic device includes a body, a plurality of pinsextending from the body, the pins including connections or couplings tofacilitate movement of a first portion of the pins relative to a secondportion of the pins, the pins being controllable to control the movementof the first portion relative to the second portion and to control aforce applied by the pins on an external object, sensors cooperatingwith the pins to detect forces externally applied to the pins, acommunication subsystem for communication, over a network, with a remoteelectronic device, and a controller coupled to the pins, the sensors,and the communication subsystem. The controller controls the electronicdevice to, based on detected forces externally applied to the pins,transmit a signal to the remote electronic device for the control of theremote electronic device, and to, based on signals received from theremote electronic device, actuate ones of the pins to control themovement of the first portion relative to the second portion and tocontrol the force applied by the pins on the external object.

A simplified block diagram of an example of an electronic device 100 fortouch translation is shown in FIG. 1. The electronic device 100 includesmultiple components, such as a main processor 102 that controls theoverall operation of the electronic device 100. The electronic device100 may be mounted to another object or device, may include mountingbrackets or geometry to facilitate mounting to another object or device,or may be in the form of a sheet for resting on a surface. According toone example, the electronic device 100 is incorporated into a case foranother electronic device such as a smartphone or tablet computer.Alternatively, the electronic device 100 may be incorporated orintegrated into another electronic device such as a portable electronicdevice, smartphone, or tablet computer. The electronic device 100 mayalso be generally transparent and may be overlaid on a display ofanother electronic device such as a smartphone or tablet computer. Thus,the display on which the electronic device 100 is overlaid may be atouch-sensitive display. Optionally, the electronic device 100 ismanually removable when not in use. Alternatively, the electronic device100 may be opaque or partially opaque and is included in a cover thatcovers part or all of the display 118 when utilized. For example, theelectronic device 100 may be incorporated into a phone case that ismanually located over the display to close the case over the displayprior to touching the electronic device 100, which may include holdingthe electronic device 100 to the face of the user.

The main processor 102 interacts with other components of the electronicdevice 100, including, for example, a temporary storage device 104, amemory 106, an auxiliary input/output (I/O) subsystem 108, acommunication subsystem 110, a power source 112, and, optionally, othersubsystems 114. Additionally, the main processor 102 interacts with acontroller 116 that is coupled to actuators 118 that are utilized tocontrol movement of pins, also referred to as fingers, about connectionsor couplings within the pins.

The actuators 118 may be, for example, linear actuators, hydraulicactuators, pneumatic actuators, magnetic actuators, or any combinationof different types of actuators that are coupled to parts of the pins tocontrol movement of one portion of a pin relative to another portion ofthe pin or to control the elevation, or distance of an end of the pinrelative to a body from which the pins extend, of one of the pinsrelative to the resting elevation of the pins. The controller 116 iscoupled to the main processor 102 and, based on signals from the mainprocessor 102, controls the actuation of the actuators 118.

The force sensors 120 are associated with the pins, for example, arelocated at the ends of the pins of the electronic device 100 to detectexternal forces that are applied to the pins, such as forces from auser's hand, finger, thumb, face, appendage, or other items held by auser applying force to the pins of the electronic device 100. The forcesensors 120 may be disposed in the pins, on the pins, under the pins, orany suitable combination of in, on, and under the pins to detect forceson the pins. Thus, an external force applied to the pins of theelectronic device 100 is detected utilizing the force sensors 120. Theactuators 118 may also be utilized to apply a force, by the pins, on anexternal object, such as the user's hand, finger, thumb, face,appendage, or other items, held by a user applying force to the pins.

The temporary storage device 104 may be, for example, Random AccessMemory (RAM) that stores data that is processed by the main processor102. The memory 106, such as flash memory, is utilized for persistentstorage. The memory 106 may be utilized to store an operating system andsoftware programs or components that are executed by the processor 102.

The optional auxiliary input/output (I/O) subsystem 108 may include aninterface through which, for example, a USB controller or otherperipheral device may be connected. Other input/output subsystems mayalso be utilized as well as other communications.

The communication subsystem 110 receives signals from a communicationdevice such as a portable electronic device, smart phone, tabletcomputer, laptop or other device (not shown) and sends signals throughthe communication device to which the electronic device 100 is coupled.Thus, for example, the signals from the force sensors 120 or othercommands from the main processor 102 may be sent via the communicationsubsystem 110. The communication subsystem 110 is also responsible forreceiving signals via the communication device for processing by themain processor 102 to cause actuation of the actuators 118, via thecontroller 116, in response to signals from the communication device.

The power source 112 may be one or more of rechargeable batteries,capacitors, inductive charging, inductive power, fuel cells, a port toan external power supply to power the electronic device 100.

The systems and subsystems that interact with the main processor 102 andare described herein are provided as examples only. Other subsystems 114may also interact with the main processor 102.

Although not shown in the block diagram of FIG. 1, the electronic device100 may optionally include other devices and subsystems. For example,the electronic device 100 may include a display device or displaydevices for displaying information such as pictures or other informationon the pins, one or more speakers for audio output, one or more camerasfor capturing images, which may include video, short-rangecommunications, proximity sensors, and other suitable devices orsubsystems.

Referring to FIG. 2, a system for touch translation including theelectronic device 100 is shown. In this example, the electronic device100 communicates with a similar electronic device 200 that is locatedremotely from the electronic device 100 by sending signals to theremotely located electronic device 200 via the communication device 202to which the electronic device 100 is coupled, through a network 250 andthrough a remotely located communication device 204. The electronicdevice 100 also receives signals from the remotely located electronicdevice 200 via the communication device 202, the network 250, andthrough the remotely located communication device 204.

As indicated above, the communication device 202 may be a portableelectronic device, smart phone, tablet computer, laptop or other devicethat is in communication with the electronic device 100 via thecommunication subsystem 110 of the electronic device 100. The electronicdevice 100 may optionally be physically coupled to the communicationdevice 202. For example, the electronic device 100 may be coupled to aback side of the communication device 202.

Similarly, the remote communication device 204 may be a portableelectronic device, smart phone, tablet computer, laptop or other devicethat is in communication with the remote electronic device 200 via acommunication subsystem of the remote electronic device 200. The remoteelectronic device 200 may optionally be physically coupled to thecommunication device 204.

The network 250 may include the internet and may include a cellularnetwork in addition to the internet or as an alternative to theinternet. Several communication devices may communicate through thenetwork 250. Other communications may also be utilized, including forexample, near field, Bluetooth®, WiFi, optical, radio, or a combinationof communications.

Thus, the electronic device 100 is operable to communicate with theremote electronic device 200. When a communication session begins,signals are transmitted from the electronic device 100 to the remoteelectronic device 200 in response to detecting an externally appliedforce on the pins of the electronic device 100. The signals are sent tothe remote electronic device 200 to control the remote electronic device200. In response to receipt of signals at the electronic device 100,from the remote electronic device 200, the actuators are controlled tocontrol movement of the pins of the electronic device 100 and forceapplied by the pins of the electronic device 100 on an external object,such as a user's hand, finger, thumb, face, appendage, or other items,held by a user applying force to the pins. Thus, a force applied by auser on the electronic device 100 is determined and, movement of thepins of the remote electronic device 200 is controlled and a resultingforce is applied by the remote electronic device 200. Similarly, a forceapplied by a remote user on the remote electronic device 200 isdetermined and a resulting force is applied by the electronic device.

A user pressing on the pins on the electronic device 100 at the sametime that a remote user presses on the remote electronic device 200,feels the return force caused by the user pressing on the remoteelectronic device 200.

A perspective view of one example of an electronic device is shown inFIG. 3A through FIG. 3C. The electronic device 100 includes a body 302in which the components illustrated in FIG. 1 are disposed. The body 302may be rigid. Alternatively, the body 302 may be flexible while stillproviding protection for the components shown in FIG. 1. The pluralityof pins 304 extend from the body 302. In the present example, the pins304 extend generally linearly away from the body 302. Each pin 304includes coupling 306 such that a distal portion 308 or outer portion ofthe pin 304 is moveable relative the proximal portion 310 or innerportion of the pin. The proximal portion 310 is coupled to the body 302while the distal portion 308 is coupled at the coupling 306 to theproximal portion 310.

For the purpose of the present example, the distal portion 308 ismoveable relative to the proximal portion 310, toward and away from thebody 302. Movement of a pin 304 and any force applied by a pin on anexternal object that is in contact with the end of the distal portion308 of the pin 304 or a cover or membrane coupled to the distal portion308 of the pin 304, is controlled by an actuator, such as a linearactuator, which may be, for example, a hydraulic actuator or pneumaticactuator. The linear actuator is coupled to the controller to therebycontrol sliding movement of the distal portion 308 relative to theproximal portion 310 of the pin 304, and force applied by the pin 304.In this example, coupling comprises a telescoping coupling and thesliding movement of the distal portion 308 relative to the proximalportion 310 is a telescoping movement.

For the purpose of this example, the pins 304 are covered by a flexible,elastic membrane 312 such as a latex, flexible PVC, CyberSkin® or acombination of flexible, elastic materials. Thus, in this example, thepins 304 and membrane 312 are part of a user interface of the electronicdevice 100.

The flexible, elastic membrane 312 may also be comprised of multiplelayers of materials. For example, the flexible, elastic membrane 312 mayinclude a first layer 314 that couples to at least some of the pins 304,for example, by a mechanical interlock with sockets in the layer or anadhesive layer that facilitates application of forces by the pins 312,away from the body 302 and toward the body 302. The flexible, elasticmembrane 312 may include a second layer 316 of, for example CyberSkin®,and a third layer 318 of, for example, a very thin latex. The very thinlatex may be replaceable.

Alternatively, heads 320 may be disposed on the ends of the pins 304 andthe heads are larger in diameter than the body of the pins 304, asillustrated in the example of

FIG. 3D and FIG. 3E. In this example, no membrane 312 is present. Theheads 320 may be a different material or materials than the pins 304.

Referring again to FIG. 3A through FIG. 3C, the plurality of the pins304 extend from the body 302, in a dense array of pins 304 that areeach, individually actuatable. In addition to being actuatable, the pins304 are depressible by an externally applied force. Such an externallyapplied force is detected utilizing the force sensors 120. The forcesensors may be coupled to the pins such that each pin is associated witha respective force sensor for detecting the externally applied force onthat associated pin.

In accordance with the present example, the pins 304 are small relativeto a human finger, thumb, hand, appendage, or face and are disposed in adense array on the body 302 such that an external force exerted, forexample, by a human finger, is exerted on a plurality of the pins 304.Thus, force may be applied to tens or hundreds of pins 304 by a user'sfinger pressing on the electronic device 100. As a result of therelatively high number and density of pins 304, a force is applied onthe pins 304, which together are moved in the shape of the finger orother object that applies the force.

Referring again to FIG. 2, when an external force is applied to the pins304, sufficient to cause the distal portion 308 of some of the pins 304to move toward the body 302, signals are transmitted to the remoteelectronic device 200 that is in communication with the electronicdevice 100. When no external force is applied to pins at the remoteelectronic device 200, the pins at the remote electronic device 200 thatcorrespond to the pins 304 to which the force is applied at theelectronic device 100, are moved. The corresponding pins at the remoteelectronic device 200 are moved by moving the distal portion of the pinsaway from the body.

Thus, the ends of the pins 304 are moved toward the body 302 at theelectronic device 100 to provide a depression in the surface thatgenerally follows the contour and surface profile of the object, such asa finger, that applied the force to the pins 304. At the remoteelectronic device 200, ends of the corresponding pins are moved awayfrom the body to form a projection that generally follows the contourand surface profile of the object that applied the force to the pins304. The projection formed at the remote electronic device 200 is formedby the pins covered by the elastic membrane, giving the generalappearance of the object that applied the force to the pins 304 at theelectronic device 100.

In this example, the distal ends 308 of the pins 304 are moved at theelectronic device 100 to form a depression in the surface of themembrane 312, and the distal ends of the pins are moved at the remoteelectronic device 200 to form a corresponding projection in the surfaceof the membrane. Thus, the pins at the remote electronic device 200 aremoved in the opposite direction as the pins at the electronic device 100to generally form an inverse profile.

When an external force is applied to the pins 304 at the electronicdevice 100, and an external force is applied to corresponding pins atthe remote electronic device 200, the corresponding pins at the remoteelectronic device 200 apply a force to the external object applying aforce at the remote electronic device 200. Similarly, the pins 304 atthe electronic device 100 apply a force on the object applying theexternal force at the electronic device 100. The force applied by thepins at the remote electronic device 200 to the external object,generally corresponds in magnitude to the external force applied to thepins 304 at the electronic device 100. The force applied by the pins 304to the external object at the electronic device 100 generallycorresponds in magnitude to the external force applied to the pins atthe remote electronic device 200.

A flowchart illustrating a method of controlling an electronic device,such as the electronic device 100 is shown in FIG. 4. The method may becarried out by software executed, for example, by the main processor 102of the electronic device 100. Coding of software for carrying out such amethod is within the scope of a person of ordinary skill in the artgiven the present description. The method may contain additional orfewer processes than shown or described, and may be performed in adifferent order. Computer-readable code executable by at least oneprocessor to perform the method may be stored in a computer-readablemedium, such as a non-transitory computer-readable medium.

A communication session is initiated 402 between the electronic device100 and the remote electronic device 200. The communication session isstarted by one or both of the electronic device 100 and the remoteelectronic device 200. To initiate the communication session, theelectronic device 100 and the remote electronic device perform ahandshake process, for example, in response to user selection of anoption to begin communicating with the remote electronic device 200. Thecommunication session is secured utilizing known secure communicationstechniques, including, for example, encryption and decryption to providesecurity in transmission.

Initiation of a communication session may also include userauthentication or identification. For example, knowledge-basedidentification, such as a passcode or a personal identification number,may be utilized. Alternatively or in addition, biometric identification,such as fingerprint, facial recognition, palm print, or geometry, orother biometric identification may be utilized. Such biometricidentification may be carried out by pressing a hand, face, or otherbody part against the interface of the electronic device 100. Acomparison is then made with stored data relating to the user'sidentification to confirm that the user is an authorized user of theelectronic device 100. The electronic device may also identify contoursof details of the hand or other body part and, optionally measuretemperature to confirm the temperature of the person for use inauthentication. Such biometric identification utilizing an imprint of ahand or other body part against the interface increases security overbiometric identification methods of other devices. The length of timethat the communication session lasts may be limited. For example, aftera threshold period of time, the communication session may bediscontinued unless authentication is repeated.

During the communication session, externally applied forces on theinterface of the local electronic device 100 are detected utilizing theforce sensors 120. In response to detecting an externally applied forceon the interface at 404, signals are transmitted to the remoteelectronic device 200 at 406 to actuate the actuators to controlmovement of portions of the pins about the connections or couplings tothereby control movement and forces applied by the pins of the remoteelectronic device 200.

Signals are also received at the local electronic device 100. Thesignals are received from the remote electronic device 200 in responseto externally applied forces that are detected at the remote electronicdevice 200.

In response to receipt of signals at the local electronic device at 408,the actuators 118 are actuated at 410 to control movement of portions ofthe pins about the couplings to thereby control movement and forcesapplied by the pins 304 of the local electronic device 100.

Because the pins 304 of the electronic device 100 include couplings 306to facilitate movement of the distal portion 308 relative to theproximal portion 310, the pins 304 are movable toward and away from thebody 302 and are operable to apply a force to an object touching thepins 304. In addition, the pins 304 are controlled to form a shape, suchas a projection, that generally follows the contours and surface profileof an object touching the interface of a remote device that is incommunication with the electronic device 100. Utilizing the movement ofthe pins 304 and force application, the electronic device 100, incooperation with a remote electronic device 200, simulates touch betweentwo people that are each utilizing a respective one of the electronicdevices.

Utilizing such electronic devices 100, 200, touch contact is simulatedto give the users the perception of touch. For example, a first userthat presses on the pins 304 of the local electronic device 100, while asecond user presses the pins of the remote electronic device 200, whichis in communication with the local electronic device 100, perceivestouch contact with the second user. For example, if both users resttheir hands on the respective interface, the first user perceives thathe or she is resting a hand on a hand of the second user.

Similarly, the second user perceives that he or she is resting the handon the hand of the first user. In another example, facial contact isperceived when the users press their faces against the interface.

Latency introduced from various sources such as transmission time,processing time, and actuation, may be of the order tens of millisecondsor greater. To reduce the problems introduced by latency and thus a lackof synchronization between the two electronic devices 100, 200, thesoftware stored, for example, in the memory 106 at each electronicdevice, is utilized to smooth out actions and reactions to generallymaintain the simulation of touch and facilitate perception of touch byeach user.

The communication session or parts thereof may optionally be recorded bystoring information relating the communication session. For example,received signals from an electronic device may be recorded by storingrelated information in the memory, such as the memory 106. For example,the signals received from a remote electronic device in response to auser placing his or her hand on the remote electronic device, may bestored and utilized later to reproduce the simulation of touch after thecommunication session has ended.

In the above-described method, the electronic device 100 enters acommunication session with the remote electronic device 200 to simulatetouch at both the electronic device 100 and at the remote electronicdevice 200. Optionally, the application of force and movement of thepins may occur at an electronic device, such as the electronic device100, independent of a communication session with another electronicdevice. Signals may then be transmitted, for example, by a socialnetworking platform, to share the signals with a recipient or withmultiple recipients. For example, a touch may be broadcast to multiplerecipients. In one example, a user places his or her lips on theelectronic device and kisses the interface. The signals resulting fromthe kiss may be stored remotely for another user or users to obtain. Forexample, a movie star may make a kiss available for a plurality of fansto receive on their own electronic devices.

In another example, signals may be received at the electronic device 100from a plurality of remote electronic devices and the signals may becombined, modified, averaged, or any combination thereof. Referring tothe example of the movie star making a kiss available for a plurality offans, the fans, in return, may make a kiss available for the movie star.Thus, a plurality of touches may be, for example, averaged to provide acombined response. Alternatively, a single, representative response,which may be from a single user, may be provided, where that responsefalls within a predetermined range of feedback.

According to another example, the electronic device 100 may be utilizedby moving the user interface and applying forces to a user to simulateinteraction with a virtual person or object.

A simplified block diagram of another example of an electronic device100 for touch translation is shown in FIG. 5. In the example shown anddescribed in FIG. 1, the electronic device 100 is utilized inconjunction with a communication device, such as a smartphone, tabletcomputer, or laptop computer, in order to communicate over a networkwith a remote electronic device. In the example shown in FIG. 5, theelectronic device 500 may be utilized without connecting to a seconddevice. Thus, the electronic device 500 in this example is operable tocommunicate over a network without the use another communication device.

Many of the elements or components referred to in FIG. 5 are similar tothe elements or components in FIG. 1. For simplicity and clarity ofillustration, the reference numerals are raised by 400 to indicatecorresponding or analogous elements.

The electronic device 500 includes multiple components, such as a mainprocessor 502 that controls the overall operation of the electronicdevice 100. As indicated, the electronic device 500 is operable tocommunicate, over a network, with a remote electronic device. Theelectronic device 500 in this example, may be any suitable size,depending on the application or intended use.

The main processor 502 interacts with other components of the electronicdevice 500, including, for example, a temporary storage device 504, amemory 506, an auxiliary input/output (I/O) subsystem 508, acommunication subsystem 510, a power source 512, and, optionally, othersubsystems 514. Additionally, the main processor 502 interacts with acontroller 516 that is coupled to actuators 518 that are utilized tocontrol movement of the pins about connections or couplings within thepins.

The functions of many of the components are similar to those describedwith reference to FIG. 1 and are therefore not described in detail againherein.

In the present example, communication functions are performed throughthe communication subsystem 110. Data received by the electronic device100 is decompressed and decrypted by a decoder 522. The communicationsubsystem 510 receives signals from and sends messages to a network (notshown).

The main processor 502 may also interact with other components such as aspeaker 524, a microphone 526, a display 528, one or more cameras 530,and short-range communications 532.

The speaker 524 outputs audible information converted from electricalsignals, and the microphone 526 converts audible information intoelectrical signals for processing. The display 528 may be any suitabledisplay or displays for displaying information, for example, on thepins. The display 528 may project an image or may be embedded in thepins or on heads that are disposed on the pins in order to displayinformation, such as images, on the pins.

The camera or cameras 530 are utilized to obtain images or video of theuser of the electronic device 500. Optionally, the cameras 530 may beutilized to obtain images or video of the user's surroundings as well.Each of the cameras includes the functional components for operation ofthe camera, including the lens, the image sensor, and, optionally, alight sensor and light source, such as infrared light emitting diodes(LEDs). The cameras may be one or more of visual light cameras, 3Dsensing cameras, light field cameras, forward looking infrared cameras,near infrared cameras, ultraviolet cameras, or other imaging devices.

The short-range communications 532 may be utilized to perform variouscommunication functions. For example, the short-range communications 532may include Bluetooth or infrared (IR) communications capability forcommunicating with another electronic device, a peripheral device, oraccessory.

Referring to FIG. 6, a system for touch translation including theelectronic device 500 is shown. In this example, the electronic device500 communicates with a similar electronic device 600 that is locatedremotely from the electronic device 500 by sending signals to theremotely located electronic device 600 via the network 250. Theelectronic device 500 also receives signals from the remotely locatedelectronic device 600 via the network 250. Thus, no communication deviceis utilized in the present example of the electronic device 500 forcommunication via the network 250. The electronic device 500 is operableto communicate directly over the network 250.

Thus, the electronic device 500 is operable to communicate with theremote electronic device 600. When a communication session begins,signals are transmitted from the electronic device 500 to the remoteelectronic device 600 in response to detecting an externally appliedforce on the pins of the electronic device 500. The signals are sent tothe remote electronic device 600 to control the actuators and therebycontrol the movement of pins and forces applied by the pins at theremote electronic device 600. In response to receipt of signals at theelectronic device 500, from the remote electronic device 600, theactuators are controlled to control movement of the pins of theelectronic device 500 and forces applied by the pins of the electronicdevice 500 on an external object, such as a user's hand, finger, thumb,face, appendage, or other items, held by a user applying force to thepins. Thus, a force applied by a user on the electronic device 500 isdetermined, movement of the pins of the remote electronic device 600 iscontrolled, and a resulting force is applied by the remote electronicdevice 600. Similarly, a force applied by a remote user on the remoteelectronic device 600 is determined and a resulting force is applied bythe pins of the electronic device 500. The operation of the electronicdevice 500 may be similar to the operation of the electronic device 100and thus, the operation is not further described herein.

The method described above and shown in FIG. 4 is also applicable to theelectronic device shown in FIG. 5. The method may be carried out bysoftware executed, for example, by the main processor 502 of theelectronic device 500. Details of the method shown in FIG. 4 anddescribed above are also applicable to the electronic device 500 and aretherefore not described again herein.

As with the electronic device 100, the pins of the electronic device 500include couplings to facilitate movement of portions of the pins and tofacilitate application of a force to an object touching the pins. Inaddition, the pins are controlled to form a shape, such as a projection,that generally follows the contours and surface profile of an objecttouching the interface of the remote device 600 that is in communicationwith the electronic device 500. Utilizing the movement of the pins andforce application, the electronic device 500, in cooperation with aremote electronic device 600, simulates touch between two people thatare each utilizing a respective one of the electronic devices. Utilizingsuch electronic devices 500, touch contact is simulated to give theusers the perception of touch.

The force that is applied by the electronic device 500 on the user orthe force that is applied by the remote electronic device 600 on theremote user is controlled such that only forces that are within apredetermined range are transmitted. For example, signals that resultfrom forces that are deemed to be outside of a safe range, for example,that may result in blunt trauma or sharp forces that may cause injuryare not transmitted or are not utilized by the receiving electronicdevice. Alternatively, such forces may be altered, for example to reducethe speed of the force, reduce the sharpness, reduce the magnitude, orany suitable combination of these alterations in force.

One or both the remote electronic device 600 and the local electronicdevice 500 may compare the force or value representative of the force toa threshold limit to determine whether the force is within predeterminedsafety limits. This method may be carried out, for example in the methodshown and described above with reference to FIG. 4. Thus, the method iscarried out by software executed, for example, by the main processor 502of the electronic device 500. Coding of software for carrying out such amethod is within the scope of a person of ordinary skill in the artgiven the present description. Additional or fewer processes may also beperformed and computer-readable code executable by the processor 502 toperform the method may be stored in memory 506.

Thus, as part of the process, for example, at 404 of FIG. 4, theprocessor 502 of the electronic device 500 may compare the force or avalue representative of the force to a threshold limit stored in memory506. In response to determining that the force or value meets or exceedsthe threshold limit, the processor 502 of the electronic device 500 doesnot transmit the associated signals to the remote electronic device 600such that the force is not applied to user of the remote electronicdevice 600. On the other hand, in response to determining that the forceor value is less than the threshold limit stored in memory 506, thesignals are transmitted and the force is applied to the user of theremote electronic device 600.

In addition, as part of the process, for example, at 408 of FIG. 4, theelectronic device 500 may compare signals received from the remoteelectronic device 600 to predetermined values prior to actuatingactuators to apply forces, by the local electronic device 500, to theuser. In response to determining that the force or value meets orexceeds the threshold limit, the processor 502 of the electronic device500 does not actuate the actuators such that the force is not applied touser of the local electronic device 500. On the other hand, in responseto determining that the force or value is less than the threshold limitstored in memory 506, the force is applied to the user of the remoteelectronic device 600.

The size or shape, such as the width across which the force is appliedmay also be utilized such that forces from very sharp objects are nottransmitted to the user. For example, the threshold limit may varydepending on the dimensions, such as width, across which the force isapplied to the pins of the electronic device 500. Thus, for example, theelectronic device 500 may maintain a lookup table in memory 506 and thethreshold limit that is utilized for the comparison is identified fromthe lookup table and is dependent on dimensions including length andwidth of the applied force.

Optionally, the output may be scaled relative to the input such that,for example, inputs provided by a baby or a person with neuromusculardamage, which are by nature relatively weak, are amplified by somefactor. Such scaling is also useful where one of the users desires touchthat is stronger or weaker than the other user normally provides.

A perspective view of another example of an electronic device is shownin FIG. 7. Although the electronic device 500 described with referenceto FIG. 5 is referred to in the present description with reference toFIG. 7, the present description is equally applicable to the electronicdevice 100 described herein with reference to FIG. 1.

The electronic device 500 includes a body 702 in which the componentsillustrated in FIG. 1 are disposed. The body 702 may be rigid.Alternatively, the body 702 may be flexible while still providingprotection for the components therein. The plurality of pins 704 extendgenerally away from the body 702. In the present example, the pins 704each include a plurality of couplings 706, which are articulatingjoints. The couplings 706 in each pin 704 may include more than one typeof articulating joint to facilitate various types of movements ofportions of the pins 704. Although three joints 706 are illustrated inthe Example of FIG. 7, fewer or more joints may be utilized tofacilitate movement of the portions of the pins 704. The articulatingjoints may include, for example, hinge joints, prismatic or slidingjoints, revolute joints, or any suitable combination of joints or othercouplings. The couplings together provide a linkage to facilitatemovement in more than one axis. Thus, the portions of the pins 704 arecoupled together about couplings to facilitate movement in alldirections, as shown in FIG. 7B. In addition, all or a subset of thepins 704 may be coupled to the body 702 utilizing a coupling, forexample, to facilitate gliding movement of the pins 704 relative to thebody 702.

Movement of the pins 704 about the couplings 706 may be controlled bywires 708, for example, that couple portions of the pins 704 to the body702 or to other portions of the pins 704, as illustrated in FIG. 7C. Thewires 708 may be controlled by actuation of the actuators 518. Forexample, the wires may be pulled when actuators 518 of the electronicdevice 500 are actuated to move the heads 714 on the pins 704 about acoupling. Optionally, some or all of the pins 704 may be disposed onrollers 712 on the body 714 to facilitate movement of the pins 704relative to the body 714, as shown in FIG. 7A and 7D.

The pins 704 are movable along the body 702 in a sliding or glidingmotion, movable, toward and away from the body 702. Different portionsof the pins 704 are also moveable relative to the body 702 to facilitatemovement of the pins 704 in other directions. Movement of the pins 704and any force applied by a pin 704 on an external object that is incontact with a head 714 on the pin 704, is controlled by multipleactuators that cooperate to control movement and force applied by thepin 704. Thus, the heads 714 on the pins 704 are movable in threedimensions facilitating flexion, extension, rotation, adduction,abduction, or any combination thereof, of the pins 704.

In this example, heads 714 are mounted on the ends of the pins 704. Theheads 714 are geometrically shaped to include a plurality of facets 716and are coupled to the pins 704 to facilitate rolling of the heads 714relative to the pins 704 to select which of the facets 716 is exposed ordirected outwardly. Thus, the head 714 may be rolled to expose any oneof, for example, four facets 716 depending on the application. Eachfacet 716 may have different material properties to provide differentsensations to the touch, as illustrated in FIG. 7E and FIG. 7F. Forexample, the four facets 716 may include one facet covered by a materialsuch as latex or CyberSkin®, a second facet covered with very fine wispsof hair or hair-like material on a silicone or material base, a thirdfacet covered by a more dense coat of hair; and a fourth facet coveredby a cloth material. Thus, depending on which of the facets 716 isexposed at the end of each of the pins 704, the heads 714 are utilizedto simulate the feel of different surfaces or textures.

As in the embodiment described herein with reference to FIG. 3, aplurality of the pins 704 extend from the body 702, in a dense array ofpins 704 that are, individually actuatable and each individual actuatorwithin the pins is actuatable. In addition to being actuatable, the pinsare depressible or flexed by an externally applied force. Such anexternally applied force is detected utilizing the force sensors 520.The force sensors 520 are disposed on the pins 704 to detect externallyapplied static or dynamic forces including, for example, compressiveforce, frictional force, tensile force, torsion, and any combination ofsuch forces.

The pins 704 are small relative to a human finger, thumb, hand,appendage, or face and are disposed in a dense array on the body 702such that an external force exerted, for example, by a human finger, isexerted on a plurality of the pins 704. Thus, force may be applied totens of pins 704, hundreds of pins 704, or more, by a user's fingerpressing on the electronic device 500. As a result of the relativelyhigh number and density of pins 704, such a force is applied on the pins704, which together generally follow the contour and surface profile ofthe object that applied the force to the pins 304.

Because, the pins 704 are moveable in multiple axes in response to anexternally applied force or in response to signals received via thecommunication subsystem 510, for example, and are operable to applyforce in multiple directions against an external object, more complextouch interaction in which forces are applied in more than one directionor plane may be simulated. In addition, further contours and movement ofthe object that applied the force to the pins 304 may be formed in asimilar, remote electronic device in communication with the localelectronic device 500.

According to one example, an electronic device 500 may be held up to thethroat of a user while a doctor manipulates a remote device that thedoctor is using to simulate the feel of the glands of the user of theelectronic device 500. Thus, the doctor manipulates the pins on theremote device such that the pins 704 apply a light force against theglands of the user. The size and contours of the glands may bedetermined by the doctor based on the reaction forces on the pins 704,which are utilized at the remote device to simulate the throat, which isthe object to which the force is applied and that is applying thereaction forces against the pins 704. With sufficient sensitivity, adoctor can also detect the pulse as the pulse is simulated at the remotedevice.

Alternatively, the electronic device 500 may be utilized to simulate aphysical handshake between remotely located users, hand holding, cheektouching, and any other suitable touch interaction.

In addition, with movement of the heads 714 relative to the body 702,for example, in a sliding or gliding motion, a rubbing or friction forcemay be simulated. To facilitate simulation of rubbing or friction, theelectronic device 500 may optionally introduce noise into signals sentto a remote electronic device or received from the remote electronicdevice such that the movement of the heads 714 relative to the user isnot smooth.

Referring again to FIG. 6, when an external force is applied to the pins704, sufficient to cause flexion or movement of the pins 704 relative tothe body 702, signals are transmitted to the remote electronic device600 which is in communication with the electronic device 500. When noexternal force is applied to pins at the remote electronic device 600,the pins at the remote electronic device 600 that correspond to the pins704 to which the force is applied at the electronic device 500, aremoved. The corresponding pins are moved by moving the heads on the pinsin an opposite direction relative to the body.

Thus, the heads 714 on the pins 704 are moved in a direction relative tothe body 702 at the electronic device 500, for example, providing adepression in the surface that generally follows the contour and profileof the object, such as a finger, that applied the force to the pins 704.At the remote electronic device 600, the heads on the corresponding pinsare moved in an opposite direction relative to the body, for example,forming a projection that generally follows the contour and profile ofthe object that applied the force to the pins 704. The projection formedat the remote electronic device 600 is formed by the heads on the pins,giving the general appearance of the object that applied the force tothe heads 714 on the pins 704 at the electronic device 500.

Thus, the shape formed by the movement of the heads 714 on the pins 704relative to the body 702 when an external force is applied to the heads714 at the electronic device 500, is the inverse of the shape formed bythe movement of the heads on the pins at the remote electronic device600. For example, a user pressing down with a hand on the heads 714 onthe pins 704 presses with the palm toward the body 702 of the electronicdevice 500. For the remote electronic device 600 in communication withthe electronic device 500, the shape that is formed follows the contoursof the hand, with the palm of the hand facing away from the body of theremote electronic device 600.

When external forces are applied to the heads 714 on the pins 704 at theelectronic device 500, and external forces are applied to heads oncorresponding pins at the remote electronic device 600, the heads on thecorresponding pins at the remote electronic device 600 apply forces tothe external object applying the forces at the remote electronic device600. Similarly, the heads 714 on the pins 704 at the electronic device500 apply forces on the object applying the external force at theelectronic device 500. The forces applied by the pins at the remoteelectronic device 600 to the external object generally correspond inmagnitude and direction to the external forces applied to the heads 714on the pins 704 at the electronic device 500. The forces applied by theheads 714 on the pins 704 to the external object at the electronicdevice 500 generally correspond in magnitude and direction to theexternal forces applied to the heads on the pins at the remoteelectronic device 600.

Utilizing the movement of the heads 714 on the pins 704 and forceapplication, the electronic device 500, in cooperation with a remoteelectronic device 600, simulates touch between two people that are eachutilizing a respective one of the electronic devices. Utilizing suchelectronic devices 500, touch contact is simulated to give the users theperception of touch.

As described above, software may be utilized to smooth out actions andreactions to generally maintain the simulation of touch and facilitateperception of touch by each user to compensate, at least in part, forlatency introduced from various sources.

In addition to simulating touch, the heads 714 may optionally beutilized to emit audio. For example, the heads 714 may be moved togetherto collectively emit audio, similar to a speaker.

The heads 714 may optionally be operable to be heated or cooled or bothheated and cooled, for example utilizing a heating or warming fluidwithin the pins 704. Alternatively, a heating filament may be disposedwithin or around each head 714 or pin 704. Utilizing a heating elementor fluid, the heads 714 may be heated, for example to about the skintemperature of the sender. In addition, a thermocouple may be includedin the pins 704 or in the heads 714 to measure the temperature of thepins 704 or the heads 714.

In addition to detecting forces and to simulating forces or objectsapplied to a remote electronic device, the heads 714 on the pins 704 orthe flexible, elastic membrane may be utilized to detect touches. Forexample, a patterned layer or layers of indium tin oxide may bedeposited on the surface of the heads or on the surface of the elasticmembrane for detecting touches thereon. For example, sensors may bedisposed on or near the outer surface of the electronic device 500 formutual-capacitance touch sensing.

Capacitive touch sensors may be used independently or in conjunctionwith other sensors to obtain input, for example to identify externalcontact with the device. For example, capacitive touch sensors may beused to distinguish between input that is a result of contact with skin,which is sensed utilizing capacitive sensors, or with a non-conductiveobject, which is not sensed utilizing capacitive sensors. The signalsprovided to the remote electronic device 600 may include suchinformation to alter the tactile sensations provided to the user of theremote device 600. For example, in response to determining that theinput is a result of contact with the skin, signals are sent to causeheating of the interface to simulate skin contact. In response todetermining that the input is not a result of skin contact, the signalssent to the remote electronic device 600 do not result in heating of theinterface. The recipient at the remote electronic device 600 may alterthe touch interaction to scale the simulated contact, for exampleamplifying or reducing the force, to change the temperature, or to makeany other suitable modification.

In another aspect, sensors, including capacitive touch sensors orproximity sensors may be utilized to modify, turn on, or turn off datatransmission, reception, or implementation. For example, the electronicdevice 500 may be utilized to transmit signals to the remote electronicdevice 600 when the electronic device 500 is not being held up to theear of the user. The electronic device may also modify the touch data ordiscontinue sending signals that result from the user holding theelectronic device 500 being held up to the user's ear, such as for voicecommunication.

Referring to FIG. 7A, pores 718 in the body 702 are distributedgenerally evenly across the surface of the body 702, between the pins714. Alternatively, pores may be concentrated in specific areas. Thespeaker 524 shown in FIG. 5 may be located in the body 702 to outputaudible information through the pores 718 and thus, at least some of thepores 718 are utilized as audio channels. Similarly, the microphone maybe located in the body 702 to receive audible information through thepores.

The pores 718 shown in FIG. 7A are pores in the surface of the body 702.Alternatively, pins that include the pores may extend from the body 702.In the example in which the pores are included in the pins, the pinsthat include the pores do not include heads 714. For example, the pinsmay extend in between four heads 714 such that the pore is disposedbetween the four heads 714. Such pores may be utilized as audiochannels. Alternatively, the pins that do not include heads 714 may beutilized as fluid conduits to express gas or liquid therefrom. The gasor liquid may be disposed in one or more reservoirs disposed in the body702 and expressed via one or more of the pins that do not include heads714.

Optionally, pores 720 may be disposed in some of the heads 714, asillustrated in FIG. 7G Some of the pores 718 are utilized to expelfluid, such as water, or to expel gas, such as air or to create suction.Thus, these pores 718 may be in communication with a reservoir, forexample, to expel gas or liquid therefrom or in communication with avessel to create a pressure difference to cause suction through thepores. Multiple pores may be disposed in each head 714 of at least someof the heads 714 to carry out various functions simultaneously.

In the example described above with reference to FIG. 7, a head 714 isdisposed on each pin 704. Rather than pores 718 disposed in the body 702of the electronic device 500, the pores may be disposed in the head 714on the pins 704. Optionally, the pins 704 may have a hollow section orfluid conduit 722 in communication with a reservoir 724 for the passageof fluid through at least part of the pins 704 and through pores in thehead 714.

Such pores may also be utilized for cleaning. For example, a cleaningfluid may, optionally be loaded into the device and expelled through thepores for cleaning the heads 714 on the pins 704 and the body 702. Inthis example, the pins 704 may move such that the heads 714 moverelative to the body 702 in more than one direction to distribute thecleaning fluid across the electronic device and for self-cleaning.

According to another example embodiment, the pins 704 are covered by aflexible, elastic membrane 812 such as a latex, flexible PVC, CyberSkin®or a combination of flexible, elastic materials, as illustrated in FIG.8A and FIG. 8B. In this example, the elastic membrane may include thepores 820 in communication with one or more hollow passages 822 or fluidconduits in the pins 704. The hollow passages 822 may be in fluidcommunication with one or more reservoirs, such as a liquid reservoir824 and a gas reservoir 826.

Alternatively, the flexible elastic membrane may be coupled to the pins704 and fluid may be pumped into areas in the flexible elastic membraneor a reservoir below the flexible elastic membrane to inflate theflexible elastic membrane, for example, to fill in areas between pins.The fluid may be warmed or cooled such that the fluid provides heat oris cool to the touch for improved simulation of touch.

Utilizing such pores 820, air or gas may be expelled, for example, tosimulate blowing of air, and air may be sucked inwardly to createsuction, for example, to simulate a kiss when a user at a remote devicehas his or her lips on the remote electronic device. The pores may alsobe utilized to detect when a person blows air onto the electronic device500, by detecting changes in air pressure, sound, or both air pressureand sound, and pores at the remote electronic device may be utilized toexpel air. The pores may also be utilized for the passage of sound, orlight. Optionally, fine hair or hair-like material may be moved throughthe pores to simulate fine wisps of hair on human skin, for example.

As indicated, the pores may also be utilized for cleaning. The cleaningfluid may be expelled through the pores for cleaning the elasticmembrane 812. The pins 704 may move relative to the body 702 todistribute the cleaning fluid across the membrane. Alternatively or inaddition, the elastic membrane 812, may be wiped clean by the user.

In the above-described embodiments, the pins 704 are generally evenlydistributed in an array across the body 702 and extend from the body702. The pins 704 may be different sizes and may include differentarticulating joints or other couplings. For example, the pins 704 may bedisposed on the body 702 such that the heads 714 of the pins 704 aredisposed in different layers relative to the body 702. The pins 704 maybe offset from each other but are disposed at different distances fromthe body. Thus, a pin may extend a greater distance from the body 702than an adjacent pin. In one example, three layers of pins may bedisposed on the body 702. The use of different layers of pinsfacilitates movement at greater depths, for example, for simulating ahandshake or a hug. The stacked heads 714 also facilitate movement ofthe heads to cause a change in volume, for example, as heads move aroundfrom a stacked position to project outwardly, laterally or otherwise.The heads 714 on the pins may also be generally stacked on each other onthe body 702.

As indicated above, the heads 714 may include displays 528 embeddedtherein or disposed thereon to display an image or images on the heads714 on the pins 704. Images may be displayed on sides of the heads 714as well as a top. When the pins are stacked, the images on the tops ofthe heads 714 and on the sides of the heads 714 provide depth to theimage. The heads 714 or portions of the heads 714 may also betransparent such that an image or images are displayable through theheads 714. Each head 714 may include a single pixel or a plurality ofpixels, similar to pixels of a liquid crystal display (LCD), forexample. Together, the pixels on the heads 714 are utilized to displayinformation, such as an image. Thus, the controller 516 and the mainprocessor 502 may be utilized to identify the location of each of theheads 714 and to coordinate the color and brightness of the pixels ofthe heads 714 to provide the image.

Alternatively, a flexible display may be utilized on the pins such thatthe pins cause movement and flexing of the display. In this example, thedisplay is disposed on the pins and is operable to display informationsuch as images.

Alternatively, the heads 714 may be a set color. Rather than displaysincorporated into the pins 704 or heads 714, images may be projectedonto the heads 714.

Images may also be displayed on the sides of the pins, to the extentthat the sides of the pins are exposed. In this case, the pinsthemselves include displays embedded therein or disposed thereon.Alternatively, images may be projected onto the pins.

The pins may include optical fibers or similar elements that transmitvisual data through the pins. Such fiber optic or similar elements maybe utilized in conjunction with pin positioning to provide image depth.

Optionally, the cameras 530 may be utilized to obtain images or video ofthe user's surroundings as well. Each of the cameras includes thefunctional components for operation of the camera. Video or images frommultiple cameras may be combined by programmatically stitching togetherthe video or images.

The cameras 530 may be utilized to obtain images of video of the userand part of the user at which contact occurs. The images or video may beprovided generally in real time or near real time.

Additionally, the pins 704 are movable relative to the body 702. Thepins 704 are movable along the body 702 in a sliding or gliding motion,and movable toward and away from the body 702. Different portions of thepins 704 are also moveable relative to the body 702 to facilitatemovement of the pins 704 in other directions. The pins 704 may be movedcloser together or farther apart on the body 702. For example, the heads714 on the pins or ends of the pins may be moved closer together by thevarious actuators controlling the couplings.

Optionally, the pins 704 may be grouped such that groups of the pins 704may move together relative to the body 702. For example, the groups ofpins 704 may be coupled to an intermediate seat or base that is coupledto the body 702. Sets or clusters of pins may swivel or pivot togetheron the base, relative to the body 702, about a point or axis. Sets orclusters of pins may also move together with the base, away from thebody 702 or toward the body 702.

The movement pins 704 is controlled programmatically to facilitate themovement of individual pins 704 together as a group and to control themovement of sets of pins 704 together. Thus, for example, when a set ofpins 704 move together on a base, relative to the body, other pins maymove to accommodate the movement of the set of pins, such that themovement of pins 704 does not interfere with movement of other pins 704.

The movement of groups of pins together also facilitates the simulationof more complex touch interactions in which forces are applied in morethan one direction or plane, by comparison to the simulation of asurface or applied force in one direction.

When grouped together, the pins, along with the base on which the pinsare disposed, may be removed and loaded on the body, for example,similar to the loading of a cartridge. Thus, pins that are worn or notworking may be replaced by replacing a cartridge that includes aplurality of the pins.

Alternatively, the heads of the pins may be replaceable, for example, inthe circumstance in which the heads wear out faster than the pins. Theheads may be detachable or decouplable and the pins and headsprogrammatically controlled such that the heads are decoupled from thepins and new, replacement heads are coupled to the pins. For example,the pins may extend into a cartridge that includes replacement heads,where a replacement head is attached, and the pins are then retracted.

In addition, cartridges of pins may be selected based on the material ormaterials on the heads on the pins. For example, a cartridge may beselected to simulate a surface of a hand or to simulate clothing. Thus,rather than having different facets on the heads on the pins, cartridgesof pins may be selected to simulate different surfaces.

The body may be any suitable size. As indicated above, the electronicdevice may be incorporated into a case for a smart phone. The body mayalso be much larger. For example, the body may be incorporated into acase or a part of a tablet computer. The body may be the size of a desk,small or large, or may be the size of a mattress. Two electronic devicesin communication with each other may also be different sizes.

In the example of FIG. 6, the electronic device 500 may be the size of adesk. The remote electronic device 600, however, may be incorporatedinto a case for a smart phone. The electronic device may be configuredto compensate for differences in size of the electronic device, forexample, to fill in parts of an object for which signals or informationis not transmitted.

A flowchart illustrating a method of controlling an electronic device,such as the electronic device 500 is shown in FIG. 9. The method may becarried out by software executed, for example, by the main processor 102of the electronic device 100. Coding of software for carrying out such amethod is within the scope of a person of ordinary skill in the artgiven the present description. The method may contain additional orfewer processes than shown or described, and may be performed in adifferent order. Computer-readable code executable by at least oneprocessor to perform the method may be stored in a computer-readablemedium, such as a non-transitory computer-readable medium.

The method is carried out during a communication session with a remoteelectronic device, for example, at 410 or between 408 and 410 in themethod of FIG. 4. The signals are received at the local electronicdevice 500, from the remote electronic device 600, in response toexternally applied forces that are detected at the remote electronicdevice 600.

Based on the signals received, the object is identified 902. Forexample, the signals received may be signals from fingers touching theremote electronic device. In this example, the electronic device 500determines that the fingers extend to the edges of the interface and arepart of a hand.

The electronic device 500 identifies, at 904, a matching file stored inmemory based on the identification of the object at 902. The matchingfile includes information for providing signals to actuators 518 tosimulate the fingers and hand of the user of the remote electronicdevice 600. For example, the electronic device 500 may identify aspecific user's hand based on identifying features including the shapeand contours of the fingers. Alternatively, the electronic device 500may identify a suitable hand by size and shape to go with the fingersidentified at 902.

In addition to actuating the actuators to simulate the portions of theobject that touched the interface at the remote electronic device 600,actuators 518 are actuated to simulate the missing parts of the objectutilizing the file identified at 904. Thus, in the example of thefingers touching the remote electronic device 600, in addition toactuating the actuators 518 to simulate the fingers at the electronicdevice 500, actuators 518 are actuated to control movement and forcesapplied by pins to simulate the hand.

Thus, the electronic device 500 is operable to add or fill in parts ofobjects. This method is particularly useful in the example in which thesizes of the electronic devices differ.

As indicated above, the body may be any suitable size. In addition, thebody may take any suitable shape. For example, the body may envelop theuser. Such a configuration is useful, for example, for simulating a hugor for virtual-reality applications. Other shapes may also be desirable,including a mattress, a chair, or other shape.

Signals sent to the remote electronic device 600 as a result of touchinteraction with the electronic device 500 may also be scaled based onthe size of each electronic device 500 and the remote electronic device600. For example, signals resulting from touch contact may alter thearea of touch contact at the remote electronic device 600. A ratio maybe set automatically based on device sizes. For example, an 8″electronic device in communication with a 4″ remote electronic device,may scale touch contact or movements or both by a factor of 2.Alternatively, scaling may be manually entered or may be determinedbased on predetermined rules. Alternatively, a smaller area of theelectronic device 500 may be utilized such that the area of theinterface of the electronic device 500 that is utilized is equivalent tothe area of the interface of the remote electronic device 600.

Optionally, sensors, such as an accelerometer or other suitable sensors,may be utilized detect movement of the electronic device, for example,when the entire electronic device 500 is being moved, for example, whilea user is holding the device in a hand or hands. The electronic device500 may also determine that no active movement is detected, for example,when the electronic device 500 is set down on a table and is stationaryfor a threshold period of time. A threshold force may be utilized todetermine whether or not to send signals to the remote electronic device600. Different thresholds may be utilized to determine whether or not totransmit signals resulting from touch interaction depending on whetherthe electronic device 500 detects movement of the entire device ordetects that the device is stationary. For example, an electronic device500 that is stationary for 10 seconds, may utilize a higher thresholdthan when the electronic device 500 detects active movement, such that agreater force is required to cause the electronic device 500 to sendsignals to cause the remote electronic device 600 to apply a force.

Another example of an electronic device is shown in FIG. 10. In thisexample, a flexible elastic membrane 1012 is coupled to pins 1004. Thepins 1004 each include a fluid conduit 1020 that extends through the pinto a reservoir 1022 in the body 1002. The fluid conduit 1020 is utilizedfor fluid communication between the reservoir 1022 and a respectivepocket 1024 between layers of the flexible elastic membrane 1012. Theflexible elastic membrane includes a plurality of pockets 1024 forreceiving fluid from the reservoir 1022. The fluid may be a gas or aliquid or both gas and liquid. Thus, fluid may be pumped into thepockets 1024 to inflate the flexible elastic membrane 1012. In thisexample, the pins 1004 are coupled to the flexible elastic membrane 1012and the flexible elastic membrane 1012 is moved by increasing ordecreasing the fluid in the pockets 1024, thereby expanding orcollapsing the pockets 1024. When a pocket expands as fluid is pumpedinto the pocket, the outer surface of the flexible elastic membrane 1012is moved outwardly, away from the body 1002. When a pocket collapses,the outer surface of the flexible elastic membrane 1012 is movedinwardly, toward the body 1002. Thus, the outer surface of the flexibleelastic membrane 1012 is moveable relative to the body 1002.

Actuators 1018 control fluid movement from the reservoir 1022 to thepockets 1024. The actuators 1018 are utilized to cause the fluid to flowalong the respective pins 1004 and thereby cause movement of the outersurface of the flexible elastic membrane 1012 relative to the body 1002.Thus, each actuator is individually controllable to control the movementof parts of the outer surface of the flexible elastic membrane 1012. Themovement of the flexible elastic membrane 1012 is controlled to simulatetouch contact. The fluid may be warmed or cooled , utilizing a heatingelement or a cooling fluid disposed in the area 1026 around the fluidconduit 1020 such that the fluid provides heat or is cool to the touchfor improved simulation of touch.

Force sensors are also associated with the flexible elastic membrane1012, for example, to detect external forces applied to the flexibleelastic membrane 1012.

The method shown in FIG. 4 and described herein is also applicable tothe electronic device shown in FIG. 10. The method may be carried out bysoftware executed, for example, by a main processor (not shown) of theelectronic device. Details of the method shown in FIG. 4 and describedabove are also applicable to the electronic device 500 and are thereforenot described again herein.

Thus, during a communication session, externally applied forces on theinterface of the local electronic device 1000 are detected and, inresponse, signals are transmitted to the remote electronic device.Signals are also received at the local electronic device 1000 inresponse to externally applied forces that are detected at the remoteelectronic device.

In response to receipt of signals at the local electronic device at 408,the actuators 1018 are actuated at 410 to control movement of portionsof the flexible elastic membrane 1012 to thereby control movement andforces applied by the flexible elastic membrane 1012.

Because the flexible elastic membrane 1012 is movable toward and awayfrom the body 302, the flexible elastic membrane 1012 is operable toapply a force to an object touching the flexible elastic membrane 1012.In addition, the flow of fluid into the pockets 1024 is controlled toform a shape, such as a projection, that generally follows the contoursand surface profile of an object touching the interface of a remotedevice that is in communication with the electronic device 1000.Utilizing the movement of the flexible elastic membrane 1012 and forceapplication, the electronic device 1000, in cooperation with a remoteelectronic device, simulates touch between two people that are eachutilizing a respective one of the electronic devices.

Yet another example of an electronic device is shown in FIG. 11. In thisexample, a flexible elastic membrane 1112 is coupled to the body 1102.The flexible elastic membrane 1112 includes a plurality of pockets 1124for receiving fluid therein. Fluid conduits 1120 extend through an uppersurface of the body 1102 to a reservoir 1122 in the body 1102. The fluidconduits 1120 are utilized for fluid communication between the reservoir1122 and respective pockets 1124 between layers of the flexible elasticmembrane 1112. The fluid may be a gas or a liquid or both gas andliquid. Thus, fluid may be pumped into the pockets 1124 to inflate theflexible elastic membrane 1112. The flexible elastic membrane 1112 ismoved by increasing or decreasing the fluid in the pockets 1124, therebyexpanding or collapsing the pockets 1124. When a pocket expands as fluidis pumped into the pocket, the outer surface of the flexible elasticmembrane 1112 is moved outwardly, away from the body 1102. When a pocketcollapses, the outer surface of the flexible elastic membrane 1112 ismoved inwardly, toward the body 1102. Thus, the outer surface of theflexible elastic membrane 1112 is moveable relative to the body 1102.

A controller controls fluid movement from the reservoir 1122 to thepockets 1124 and from the pockets 1124 to the reservoir 1122. Thecontroller is utilized, in conjunction with valves, to cause the fluidto flow through the fluid conduits 1020, which include apertures in asurface of the body 1102 and thereby cause movement of the outer surfaceof the flexible elastic membrane 1112 relative to the body 1102. Thus,the controller, which may include valves, for example, controls themovement of parts of the outer surface of the flexible elastic membrane1112. The movement of the flexible elastic membrane 1112 is controlledto simulate touch contact. The fluid may be warmed or cooled such thatthe fluid provides heat or is cool to the touch for improved simulationof touch.

Force sensors are also associated with the flexible elastic membrane1112, for example, to detect external forces applied to the flexibleelastic membrane 1112.

The method shown in FIG. 4 and described herein is also applicable tothe electronic device shown in FIG. 11. The method may be carried out bysoftware executed, for example, by a main processor (not shown) of theelectronic device. Details of the method shown in FIG. 4 and describedabove are also applicable to the electronic device 500 and are thereforenot described again herein.

Thus, during a communication session, externally applied forces on theinterface of the local electronic device 1100 are detected and, inresponse, signals are transmitted to the remote electronic device.Signals are also received at the local electronic device 1100 inresponse to externally applied forces that are detected at the remoteelectronic device.

In response to receipt of signals at the local electronic device at 408,the controller controls movement of portions of the flexible elasticmembrane 1112 to thereby control movement and forces applied by theflexible elastic membrane 1112.

Because the flexible elastic membrane 1112 is movable toward and awayfrom the body 1102, the flexible elastic membrane 1112 is operable toapply a force to an object touching the flexible elastic membrane 1112.In addition, the flow of fluid into the pockets 1124 is controlled toform a shape, such as a projection, that generally follows the contoursand surface profile of an object touching the interface of a remotedevice that is in communication with the electronic device 1100.Utilizing the movement of the flexible elastic membrane 1112 and forceapplication, the electronic device 1100, in cooperation with a remoteelectronic device, simulates touch between two people that are eachutilizing a respective one of the electronic devices.

Referring to FIG. 12 and as indicated above, pins 1204 may be disposedon a body 1202 such that the heads 1214 of the pins 1204 are disposed indifferent layers relative to the body 1202. Thus, the heads 1214 on thepins are generally stacked on the body 1202. Stacking of heads 1214facilitates detection of forces and movement at greater depths, forexample, for simulating a handshake or a hug. The stacked heads 1214also facilitate movement of the heads to cause a change in volume, forexample, as heads move around from a stacked position to projectoutwardly, laterally or otherwise. Images may be displayed on sides ofthe heads 1214 as well as a top. When the pins are stacked, the imageson the tops of the heads 1214 and on the sides of the heads 1214 providedepth to the image.

The pins 1204 are grouped such that groups of the pins 1204 movetogether relative to the body 1202. For example, outer groups of pins1230 are disposed on intermediate groups of pins 1232, which aredisposed on inner groups of pins 1234. In this example, the pins 1204 ofthe outer groups of pins 1230 are smaller than the pins 1204 of theintermediate groups of pins 1232 and the pins 1204 of the intermediategroups of pins 1232 are smaller than the pins 1204 of the inner groupsof pins 1232. Thus, a plurality of pins of an outer group of pins 120 isdisposed on one of the intermediate pins 1232. Similarly, a plurality ofpins of an intermediate group of pins is disposed on one of the innerpins 1234.

Movement of one of the inner groups of pins 1234 results in movement ofthe associated intermediate groups of pins 1232 and the associated outergroups of pins 1230. The outer groups of pins 1230, the intermediategroups of pins 1232, and the inner groups of pins 1234 includerespective couplings or joints facilitating movement of the pins inthree dimensions. Thus, the groups, also referred to as clusters of pinsmay swivel or pivot together relative to the body 120. The movement ofthe pins 1204 is controlled programmatically to facilitate the movementof individual pins 1204 together as a group and to control the movementof groups of pins 1204 together.

The movement of groups of pins facilitates the simulation of complextouch interactions. In addition, the pins 1204 may move to form complexshapes, such as the chair 1300 illustrated in FIG. 13.

The described embodiments are to be considered as illustrative and notrestrictive. The scope of the claims should not be limited by thepreferred embodiments set forth in the examples, but should be given thebroadest interpretation consistent with the description as a whole. Allchanges that come with meaning and range of equivalency of the claimsare to be embraced within their scope.

What is claimed is:
 1. An electronic device, the device comprising: abody; a plurality of pins extending from the body, the pins includingcouplings to facilitate movement of a first portion of the pins relativeto a second portion of the pins, the pins being controllable to controlthe movement of the first portion relative to the second portion and tocontrol a force applied by the pins on an external object; sensorscooperating with the pins to detect forces externally applied to thepins; the sensors configured to read biometric information received bypressing a portion of the user's body against one or more of thesensors; the device further configured to compare the biometricinformation against biometric information of one or more permittedusers; where the device actuates a communication subsystem forcommunication, over a network, with a remote electronic device onlywhere the biometric information has been authenticated as correspondingto the one or more permitted users; a controller coupled to the pins,the sensors, and the communication subsystem to: based on detectedforces externally applied to the pins, transmit a signal to the remoteelectronic device for the control of the remote electronic device;receive response signals from the remote electronic device; based on theresponse signals received from the remote electronic device, actuateones of the pins to control the movement of the first portion relativeto the second portion and to control the force applied by the pins onthe external object.
 2. An electronic device, the device comprising: abody; a plurality of pins extending from the body, the pins includingcouplings to facilitate movement of a first portion of the pins relativeto a second portion of the pins, the pins being controllable to controlthe movement of the first portion relative to the second portion and tocontrol a force applied by the pins on an external object; sensorscooperating with the pins to detect forces externally applied to thepins; the device in communication with a second electronic devicesubstantially similar to the electronic device, where the displacementof the sensors on the electronic device cause the electronic device tosend instructions to the second electronic device as to movement of pinson the second electronic device that represent a movement of pins on thesecond electronic device that correspond to the movement detected by thesensors disposed on the electronic device.
 3. An electronic device, thedevice comprising: a body; a plurality of pins extending from the body,the pins including couplings to facilitate movement of a first portionof the pins relative to a second portion of the pins; sensorscooperating with the pins to detect forces externally applied to thepins; the electronic device in communication with a second electronicdevice equipped with means to deform its surface to substantially matchthe shape detected by the sensors.
 4. The electronic device of claim 3being applied to a portion of the body comprising glands and actuatingtransmission of data to the second electronic device, configured toallow palpation of the second electronic device after being configured,utilizing the data, to match that portion of the body to which theelectronic device is applied.